Cluster databases provide location transparency to data by allowing multiple systems to serve the same database. One specific type of cluster database is the Oracle Real Application Clusters product, licensed by Oracle Corporation, Redwood Shores, Calif. Sets of two or more computers are grouped into real application clusters. The clusters harness the processing power of multiple interconnected computers to provide a single robust computing environment. Within each cluster, all nodes concurrently execute transactions against the same database to synergistically extend the processing power beyond the limits of an individual component. Upon the mounting of the shared database, the real application cluster processes a stream of concurrent transactions using multiple processors on different nodes. For scale-up, each processor processes many transactions. For speed up, one transaction can be executed spanning multiple nodes.
Cluster databases provide several advantages over databases that use only single nodes. For example, cluster databases take advantage of information sharing by many nodes to enhance performance and database availability. In addition, applications can be sped up by executing across multiple nodes and can be scaled-up by adding more transactions to additional nodes. Multiple nodes also make cluster databases highly available through a redundancy of nodes executing separate database instances. Thus, if a node or database instance fails, the database instance is automatically recovered by the other instances, which combine to serve the cluster database.
Cluster databases can be made more highly available through integration with high availability frameworks for each cluster. The inclusion of these components provides guaranteed service levels and ensures resilient database performance and dependable application recovery. Organizationally, individual database servers are formed into clusters of independent interconnected nodes. Each node communicates with other nodes using the interconnection. Upon an unplanned failure of an active database server node, using clusterware, an application will fail over to another node and resume operations, without transaction loss, within a guaranteed time period. Likewise, upon a planned shutdown, an application will be gracefully switched over to another node in an orderly fashion.
The guarantee of service level thresholds is particularly crucial for commercial transaction-based database applications, such as used in the transportation, finance, and electronic commerce industries. System downtime translates to lost revenue and loss of market share. Any time spent recovering from a system failure is measurable in terms of lost transactions. Consequently, high availability systems budget a set time period to help minimize lost revenue due to unplanned outages. High availability systems also budget for planned service interruptions.
Table 1 describes the effects of service outages on a TCP/IP-based client. In the first case, an outage with sockets closed due to software failure or node shutdown, the client receives an error and recovers. In the second case, an outage with sockets left open, the client blocks and waits from 75 seconds to two hours.
TABLE 1Client Effects.State of SocketsConversation (SQLBlocked in I/O ReadAfter OutageConnection Requestor PL/SQL Request)or WriteSocket ClosedClient receives errorClient receives errorClient receives error(software failure ornode shutdown)Socket left openTcp_ip_abort_cintervalTcp_ip_abort_intervalTcp_keepalive_interval(node panics)(75 seconds)(10 minutes)(2 hours)
In the prior art, highly availability database applications provide one example of a form of high availability application. Other forms of general high availability applications relate analogously. High availability database applications are typically implemented by building an infrastructure for each database instance executing on a single node. This type of implementation is termed single instance failover. Single instance failover solutions depend upon both fast failure detection and the full relocation of server or node resources within the allotted time recovery period. Upon detecting a database instance failure, the database instance is restarted on a spare node of the service cluster and all resources are moved to the new node to allow the spare node to complete the recovery. Database instance failure is detected through polling performed by monitors external to the database instance or via daemon processes operating as shell scripts in user memory space. Examples of prior art systems that implement single instance failover solutions include MC Service Guard, licensed by the Hewlett Packard Co., Palo Alto, Calif.; Sun Clusters, licensed by Sun Microsystems, Inc., Palo Alto, Calif.; HACMP, licensed by IBM, Armonk, N.Y.; and CAA, licensed by Compaq Computers, Austin, Tex.
The approach taken by these single instance failover solutions is inherently serial. A typical failover has a mean time to recover of about three to five minutes, an unsatisfactorily long period of time for most production databases. Time is lost in detecting, validating, and recovering from the failure. Moreover, an external monitor or daemon process can take 30 seconds or more to detect an application failure. Additional time is then lost in taking appropriate corrective action, including stopping the failed database instance, relocating the resources the failed database instance requires to a spare server, and restarting the database instance and high availability monitors on the new server. Even under the best circumstances, a failover and recovery can take several minutes.
Therefore, there is a need to improve time to recover in a high availability cluster database environment. Such an approach would provide higher system availability and faster application restart in the event of system failure or loss of database access. Such an approach should allow the recovery of failed nodes to proceed in parallel and off the critical path of application restart, while other processing resumes substantially uninterrupted on other surviving nodes.
There is a further need for an approach to structuring clustered database instances groups for high availability, where each include one or more dependent systems configured to take over in the case of a failover or switchover event from one or more of the other cluster members. Preferably, such an approach should also enable dynamic runtime load balancing.